Applying the Pyramid Method in DUC 2005
نویسندگان
چکیده
In DUC 2005, the pyramid method for content evaluation was used for the first time in a crosssite evaluation. We discuss the method used in creating pyramid models and performing peer annotation. Analysis of score averages for the peers indicates that the best systems score half as well as humans, and that systems can be grouped into better and worse performers. There were few significant differences among systems. High score correlations between sets from different annotators, and good interannotator agreement, indicate that participants can perform annotation reliably. We found that a modified pyramid score gave good results and would simplify peer annotation in the future.
منابع مشابه
Applying the Pyramid Method in the 2006 Document Understanding Conference
The pyramid evaluation effort for the 2006 Document Understanding Conference involved twenty-two sites on twenty document sets. Each pyramid content model (one per document set) was constructed from four human summaries. Peer systems were scored using the modified pyramid score introduced in DUC 2005. ANOVAs with score as the independent variable and nine factors yielded three significant facto...
متن کاملERSS 2005: Coreference-Based Summarization Reloaded
We present ERSS 2005, our entry to this year’s DUC competition. With only slight modifications from last year’s version to accommodate the more complex context information present in DUC 2005, we achieved a similar performance to last year’s entry, ranking roughly in the upper third when examining the ROUGE-1 and Basic Element score. We also participated in the additional manual evaluation base...
متن کاملCLASSY Query-Based Multi-Document Summarization
Our summarizer is based on an HMM (Hidden Markov Model) for sentence selection within a document and a pivoted QR algorithm to generate a multi-document summary. Each year, since we began participating in DUC in 2001, we have modified the features used by the HMM and have added linguistic capabilities in order to improve the summaries we generate. Our system, called “CLASSY” (Clustering, Lingui...
متن کاملFormal and functional assessment of the pyramid method for summary content evaluation
Pyramid annotation makes it possible to evaluate quantitatively and qualitatively the content of machine-generated (or human) summaries. Evaluation methods must prove themselves against the same measuring stick – evaluation – as other research methods. First, a formal assessment of pyramid data from the 2003 Document Understanding Conference (DUC) is presented; this addresses whether the form o...
متن کاملEvolving XML Summarization Strategies in DUC
In the Document Understanding Conference for 2005, CL Research made some improvements in its summarization routines based on the use of massively XML-tagged documents containing increasingly richer characterizations of texts. We extended the Knowledge Management System to include an improved capability for identifying redundancy when adding extracted sentences to the summary and for improving t...
متن کامل